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Abstract 

This paper introduces a novel Markov process formulation 
of stochastic fault growth modeling, in order to facilitate the 
development and analysis of prognostics-based control adap- 
tation. A metric representing the relative deviation between 
the nominal output of a system and the net output that is ac- 
tually enacted by an implemented prognostics-based control 
routine, will be used to define the action space of the formu- 
lated Markov process. The state space of the Markov pro- 
cess will be defined in terms of an abstracted metric repre- 
senting the relative health remaining in each of the system’s 
components. The proposed formulation of component fault 
dynamics will conveniently relate feasible system output per- 
formance modifications to predictions of future component 
health deterioration. 

1. Introduction 

Continuous improvements in fault diagnostic and fault 
growth prognostic technologies have spawned a prolific 
growth in the application domain for these technologies, and 
a growing research interest in the development of techniques 
for using available diagnostic and prognostic information to 
improve system control and maintenance. Presently, a clear 
analytical process may be applied to implement and evalu- 
ate the effectiveness of tools for fault classification and fault 
growth prediction; however, the same cannot generally be 
said for the follow-on task of making intelligent control deci- 
sions based on available diagnostic and prognostic informa- 
tion. In general, the application of an analytical approach to 
the implementation and evaluation of prognostics-based deci- 
sion making techniques will be complicated by the potential 
for high uncertainty in estimating the future effects of avail- 
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able control actions, and the need to define a computationally 
tractable space of present and future control decisions to be 
optimized over. 

Several recent publications have suggested metrics for quan- 
tifying prognostic uncertainty and evaluating the ability of 
control modifications to affect meaningful change on fault 
growth predictions (Saxena et al., 2008; P. Wang et al., 2012; 
Edwards et al., 2010). This paper describes a process for ab- 
stracting uncertain models of environmental and fault growth 
dynamics into the generalized notation of a non-deterministic 
Markov process, in order to promote an application indepen- 
dent analysis of prognostics-based control strategies. 

Markov decision processes have been widely applied to the 
representation of problems involving decision making in 
the presence of uncertain or stochastic modeling informa- 
tion in the contexts of economics (Hauriea & Moresino, 
2006), supply chain management (Parlara et al., 1995), sched- 
uled maintenance (Smilowitz & Madanat, 1994), health care 
(Sonnenberg & Beck, 1993), and many other disciplines, in 
addition to being a widely used tool for describing fault- 
adaptive and robust control problems (Zhang & Jiang, 2008). 
A formal description of fault growth modeling and remain- 
ing useful life (RUL) estimation in terms of Markov process 
models, as well as a survey of similar stochastic modeling 
techniques, are given in Banjevic and Jardine (2006). 

The state transition probabilities in a Markov process descrip- 
tion of fault dynamics may be chosen to approximate an an- 
alytical formulation of a stochastic process, such as a Gaus- 
sian process model of fault growth dynamics, as described 
in Sankararaman et al. (2009); alternatively, state transition 
probabilities may be defined purely based on empirical ob- 
servations of the fault growth process, as is the case with hid- 
den Markov model learning techniques (Baruah & Chinnam, 
2005), or they may be derived from a combination of a priori 
and empirical information. 
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The derivation of stochastic fault growth models for a partic- 
ular application is not specifically addressed in this publica- 
tion; however, an effort is made to clearly identify how vari- 
ous sources of uncertain modeling information would be in- 
corporated into the Markov process representation of fault dy- 
namics. Arguments are provided for defining the state space 
of the fault growth process in terms of a metric represent- 
ing the relative health of system components, as well as for 
defining the action space of the fault growth process in terms 
of a metric representing the relative deviation between the 
system’s nominal output response and the net system output 
that is actually enacted by an implemented prognostics -based 
control routine at each control time-step. Operational con- 
straints on minimum acceptable system output performance 
and maximum acceptable fault growth risk will be formulated 
as bounds on the action space available to implemented con- 
trollers. 

2. Generalized stochastic modeling of the 

FAULT GROWTH PROCESS 
Consider a discretized fault growth process of the form: 

7 i&Si, Si = {ai,ai+Ai,ai+2A h ...,ai+miAi} (1) 

p\ j{x (k) , a (, k ) , w (. k))=P(ji (k+l) = Sj\ 

7i (k)=Si,a = a(k) ,w = w(k)), (2) 

Si , Sj G Si, a ( k ) G A ( k ) , w ( k ) G W (k) , k G N 

m 

Yl pl i,J = 1 ’ Vi G {0,1,..., m) (3) 

3=0 

where 7 1 (k) is a random variable representing the magnitude 
of the Z th component fault mode at time-index k. Si represents 
a uniformly quantized state space of potential fault magni- 
tudes, and p\ : j(x, a, w) represents the probability of transi- 
tioning from damage state s, ; to damage state Sj, given esti- 
mates of the system state x, a set of low-level control com- 
mands a, and the states of environmental and other exoge- 
nous inputs to the system w. Eq. (3) specifies that the sum of 
all transition probabilities defined at each system state must 
always be equal to one. 

The Markov process notation given in Eqs. (l)-(3) may be 
used to describe fault growth processes for all systems in 
which the following assumptions are satisfied: 

Assumption 1. State transition probabilities p\ • are only 
dependent on the current states of 7 j, x, a , and w, and not 
on any states or inputs occurring previously. This is referred 
to as the memoryless assumption, or the Markov assumption. 
For cases in which the fault growth process is not completely 
memoryless, but future states only depend on a finite number 
m of previous states, the Markov assumption can be satisfied 
by defining the state space of the process to be the ordered m- 


tuple of the current state and the m previously visited states 
(H. S. Wang & Chang, 1996). 

Assumption 2. State transition probabilities are considered 
to be time invariant; although, it may be the case that fault 
growth models are not precisely known a priori and must 
be adapted online using techniques such as particle filtering 
(Orchard et ah, 2008) or Bayesian learning (Saha et ah, 2009). 
Online adaptation of the prognostic model on the basis of past 
observations will technically violate Assumption 1 ; however, 
the error induced by this effect is typically ignored because 
model adaptation transients are generally difficult to charac- 
terize and they will die out as the model is adapted. 

Assumption 3. At all discrete time-steps, the state space 
S', the action space A , and the space of environmental and 
other exogenous inputs to the system W are adequately rep- 
resented by a finite set of states, which will be bounded from 
above by the availability of computational resources. In the 
event that fault growth must be modeled as a continuous time 
process, such as the Poisson process (Shetty et ah, 2008), a 
representation of fault growth modeling similar to that given 
in Eq. (2) may be expressed in terms of a continuous time 
Markov process (Seifozo, 1979) or a semi-Markov process 
(Dong & He, 2007). 

The required assumptions are mild enough to allow a wide 
array of fault growth processes to be described in terms of 
the Markov process notation given in Eqs. (l)-(3) (Guidaa & 
Pulcini, 2011; Tang et ah, 2009). 

2.1. Formulating the Markov process in terms of compo- 
nent health rather than fault magnitude 

Component fault magnitudes can generally be described by a 
real number corresponding to a measurable physical property 
such as crack length, spall width, or pitting depth; although, 
in many cases, faults cannot be directly measured in situ and 
diagnostic routines are needed to approximate current fault 
magnitudes based on the secondary effects observed in avail- 
able sensor measurements. Sensor noise and modeling uncer- 
tainties will often result in significant diagnostic uncertainty, 
and it is common practice for diagnostic estimates to be re- 
ported in terms of a probability distribution over the potential 
fault magnitudes that could correspond to a given set obser- 
vations. 

The notation introduced in this subsection will add a layer 
of abstraction to the fault magnitude estimates produced by 
online fault diagnostic and fault growth prognostic routines. 
Rather than formulating the Markov process in terms of com- 
ponent fault magnitudes, a transformation is assumed to exist 
that will express the fault growth process in terms of an ap- 
plication independent metric representing the state of heath 
(SOH) of each system component. 

The SOH for component l at time t will be represented in 
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terms of a probability distribution over a uniformly quantized 
set of component health percentages between 0% and 100%; 

7i (t) GS, S = {Sl, S 2 , S m }, 

si = 0%, Si = Si— i + A%, 1 < i < 100%/ (4) 

where A% is a quantization step-size for the state space of 
7 1 . The notation given in Eq. (2) is now considered to define 
the probability of transitioning from one component SOH to 
another, on the basis of the current states of at, a, and w at 
time-index k. 

Component fault magnitudes are generally expected to mono- 
tonically increase with time; correspondingly, component 
SOH should monotonically decrease with time. A mandate of 
monotonically decreasing component health is incorporated 
into the Markov process notation as follows: 

m 

Pij = 0, if j > i, Y,Pi,i = 1 G {°’ 1 > 
o = o 

This constraint will be problematic for techniques that model 
process uncertainty with an analytical distribution that lacks 
an explicit lower bound. For example, in the case of fault 
growth prognostic techniques such as Kalman filtering and 
Gaussian process modeling, an assumption of Gaussian un- 
certainty will introduce some probability that the fault mode 
does not monotonically increase. In such cases, it would be 
necessary to assure that the probability attributed to the non- 
realizable outcomes, P ( 7 ; (r) < 7 1 (t)) for r > t, will be 
acceptably small. It will not always be the case that com- 
ponent health is strictly monotonically decreasing; however, 
consideration of this constraint serves to illustrate the flexibil- 
ity provided by representing modeling uncertainties in terms 
of a quantized probability mass function fpmf), when com- 
pared to techniques that assume a continuous probability dis- 
tribution function (pdf). 

2.2. Formulating the action space of the fault growth pro- 
cess in terms of commanded and applied loads 

Consider a process model for component damage accumula- 
tion that is expressed in terms of a metric representing the 
load or stress applied to a component at each control time- 
step: 

P l i,j{ u i) = P(n ( fc + 1 ) = s il7z (■ k) = Si,ui=ui (k)), 

Si,Sj G S, Ui € Ui (fc) (6) 

where ui represents the load applied to component l and 
Ui ( k ) represents the domain of feasible component load al- 
locations for component l at time-index k. The component 
loading variable may represent pressure, force, torque, or a 
wide variety of other stressors that drive component damage. 

Determination of Ui (k) at present and future control time- 


steps will require a mapping function to translate system 
hardware limitations and estimated environmental loading 
conditions into the component loading domain. A mapping 
between available low-level control actions and feasible com- 
ponent loadings, as well as an inverse mapping are both as- 
sumed to be known: 

F (a (k) , at (k) , 7 ( k ) , w (k)) : A(k) U ( k ) 

-1 ( 7 ) 

F (u(k) ,x(k) ,~f(k) ,w(k)) : U (k) -A A(k) 

where F represents a mapping from a given system state, 
health state, exogenous demand state, and a set of low-level 
control actions available at time-index k, onto the domain of 
feasible component load allocations available at time-index 

k. 

2.2.1. Performance constraints 

In addition to being defined in terms of the hardware limita- 
tions of control effectors, the domain of feasible component 
load allocations will generally also be bounded by operational 
constraints on minimum allowable output performance and 
maximum allowable fault growth risk. 

Consider a constraint on minimum allowable system perfor- 
mance that is defined in terms of a maximum allowable devi- 
ation from some commanded system state: 

|x c (k) - x(fc)| i < Aj (k) , i e {1,2, (8) 

where x and x c are n dimensional vectors that represent the 
actual and commanded output states of a system respectively, 
and A* specifies a maximum acceptable error between the i th 
dimensions of x and x c . 

If the system’s kinematics are known, then Newton’s laws of 
motion can be applied to express the system’s dynamics in 
terms of the instantaneous forces on its constituent compo- 
nents, 

x = f{x,u,w) (9) 

If the system is overactuated, then formulating the sys- 
tem’s dynamics in terms of instantaneous component loads 
will result in actuation redundancies being identified by the 
nullspace of u. Consider the following generic representa- 
tion of nonlinear system dynamics: 

x = A (x, w) + B (x) u (10) 

where A(x,w) € K", B (x) € R" X9 , x(t) € ffi", and 

u (t) £ K 9 . If B (x) does not have full column rank, i.e, 
rank{5 (at)} = p < q Vat, then the system is overactuated, 
and B (at) can be factorized as: 

B (at) = B v (x) B u (x) (11) 

where B„ (at) € R" xp and B u (at) 6 W x i both have rankp. 
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The system can now be rewritten as: 

x= A (x, w) + B v (x) v 
v = B u (x) u 

where u(t) £ M p represents the net output control effort pro- 
duced by the system’s q components. 

Inverting the dynamics given in Eq. (12) enables the perfor- 
mance constraint given in Eq. (8) to be expressed as a max- 
imum allowable deviation from a given output control effort 
profile: 

r = B v (x c )~ 1 j (x c - A (x c , w)) 

(fe) - n (fc)| < Aj (k) , i £ {1, 2, 

where v and r are p dimensional vectors that represent an ac- 
tual and a desired net output force to be exerted by the system 
at a given time, and Aj (k) specifies a maximum acceptable 
error between the i th dimensions of v and r. 


health 



Figure 1. Uncertainty in fault growth predictions for high and 
low loads at various prognostic horizons 


components. In such cases, the production of prognostic es- 
timates with the highest degree of realism will require future 
component loadings to be modeled as a stochastic process 
that accounts for the statistics of all parameters affecting com- 
ponent load allocations within the controlled system. 


Because B u ( x ) has full column rank, a system response, x, 
is uniquely determined by however, if the system is 

overactuated, then the allocation of load among functionally 
redundant components may be specified by minimizing the 
aggregate component damages corresponding to load alloca- 
tions in the nullspace of B u (x). Prognostics-based control 
in terms of component load allocations has been analyzed 
for an overactuated electro-mechanical actuator and an un- 
manned ground vehicle in previous publications (Bole et al., 
2010,2011). 

2.2.2. Prognostic constraints 

Prognostic constraints are typically specified in terms of a 
lower bound on the failure time of system components. Con- 
straints on the minimum acceptable component failure time 
may be specified in terms of a maximum acceptable prob- 
ability that the component will reach 0% health by a given 
time: 

P (7 1 (' *m ) = 0 %| 7 1 ( t p ) , w ( t p ) , ui ( t p )) < on (14) 

where t p is the time at which the fault growth prediction is 
made and oti is an upper bound on the probability that com- 
ponent l is failed at time £m- 

If U[ was known over the domain t = [t p , then for- 
ward induction could be used to evaluate Eq. (14) from Eq. 
(6). Many publications on the topic of prognostics-based con- 
trol opt to simplify the prognostics problem by assuming that 
component loadings will be unvarying over the prediction 
horizon: 

u(t)=u(t p ) Vt£[t p ,t M ] (15) 

However, in most cases, time-varying environmental load- 
ing conditions and time-varying component health estimates 
are expected to result in time-varying loadings on a system’s 


Consider the drawing in Figure 1, the growth of uncertainty 
in component health estimates at prognostic horizons of in- 
creasing length is shown for two potential loading conditions, 
denoted as ‘high load’ and ‘low load’. As shown in the figure, 
fault growth estimates at different loading conditions are ex- 
pected to diverge over time. In cases where prognostic uncer- 
tainty becomes very large, due to high uncertainty on predic- 
tions of future component loading conditions, or high uncer- 
tainty on predictions of component damage as a function of 
loading profiles, the magnitude of prognostic uncertainty may 
be managed by limiting the length of the prediction horizon. 

The specification of a lower bound on acceptable value at 
risk (VaR) assessments of system health over the range t = 
[ip Am] may be used to enforce constraints on component 
longevity, using fixed horizon prognostic predictions. The 
VaR of a random variable X at a confidence level ip is de- 
fined as: 

VaR v, (X) = inf {x £ R : P {X < x) > tp} (16) 

A finite horizon prognosis constraint is written as follows: 

VaRft (7j (t p + t)) >Ci(tp + r) (17) 

where r specifies a time horizon at which prognostic con- 
straints will be evaluated, ( ; specifies a lower bound on the 
depletion of component health over the range t = \t p . f ; vf], 
and fli defines the maximum acceptable probability that the 
health of component l is less than Q at time t p + r . 

An example definition of Q (t) is given in Figure 1. The spec- 
ification of an appropriate profile for (1 (f) will be left as a 
design choice. Some general considerations for the specifica- 
tion of an appropriate profile include: 

1. If component health is expected to be monotonically de- 
creasing, then £/(£) must also be a monotonically de- 
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creasing function. Additionally, if it is ever the case that 
VaRpt ( 7 1 ( t p )) < (1 ( t p + r), then the prognostic con- 
strain is unsatisfiable. 

2. The greater the difference between VaRp t ( 7 j (■ t p )) and 
0 (tp + T )- the greater the control freedom allowed un- 
der the prognostic constraint. Online or a priori infor- 
mation could be used to make adjustments to Q (t) so as 
to allow for greater control freedom during higher pri- 
ority time-periods at the expense of potentially incurring 
greater component health deterioration over those time- 
periods. 

2.3. Fault prediction in terms of relative deviations from 
nominal system outputs 

Consider the existence of a nominal control system, which 
would adequately control a system in the absence of any com- 
ponent degradation modes. This section introduces a met- 
ric to represent the relative deviation between the net output 
control effort that would have been exerted by the system on 
its environment if a nominal control law were used, and the 
net output control effort that is actually exerted by a given 
prognostics-based control routine at each control time-step. 

The proposed metric is defined for each of the system’s output 
degrees of freedom as: 

/ ) i ( fc) = M ;e {i , 2 ,..,p } (is) 

where u* and v represent the net output control effort that 
would have been commanded by a nominal control law and 
the net control effort output that is actually commanded by an 
implemented control routine at time-index k. 

The Markov process defined in Eq. ( 6 ) is rewritten in terms 
of this new metric as: 

P l s z , Sj (p(k)) = p (-n (fc+l) = Sjh (k) = Si,p = p(k)) 

= J P(v r (k) | p r = p r (k)) ■ (19) 

v r e N r {k) 

P(li (k+ 1)= Sjl'yi (k) = Si, u (k)=H(v (k))) du 
Si,Sj G S,ui G Ui (k) , r G {1,2, 

where P(v r {k)\p r ) can be estimated using available 
stochastic modeling of the future net output control effort 
demands on the system, and H (v (k)) represents a map- 
ping from a net system output force vector to a component 
loading vector. As described in Section 2.2.1, if no over- 
actuation is present in the system, then component loadings 
are uniquely specified by a net system output force profile; 
however, if the system is overactuated, then the nullspace of 
the component loading domain can be resolved by an opti- 
mization routine that seeks to minimize the aggregate loss 


of health among functionally redundant degrees of freedom. 
Note that all modeling of internal and external dynamics that 
affect component degradation are now described by the prob- 
ability transition matrix p' s s (p). This formulation of fault 
growth dynamics provides a convenient means for analyzing 
the prognostics-based control problem, because it directly re- 
lates modifications to system output performance to predic- 
tions of component degradations. 

The performance constraint, defined in terms of allowable net 
system output control effort in Eq. (13), is now expressed in 
terms of p as: 

|1 - Pi (k)\ < A* (k) , i G {1,2, (20) 

where A, defines a constraint on the maximum allowable de- 
viation from a system’s nominal control effort output in di- 
mension i at time-index k. 

A uniformly quantized state space for p, under the perfor- 
mance constraint is defined as: 

Pi (k) G 0(fc) , 0(fc) = [9 1 ,d 2 ,...,0 m \, 

0 i = 1 — A* (k) , 9 m = l + A i(k) ( 21 ) 

2.3.1. An example of output control effort regulation 

A simple example is considered here to examine the regula- 
tion of a system’s net control effort output using the perfor- 
mance metric p. The example system to be regulated is a 
linear actuator attached to a simple mass-spring-damper sys- 
tem, defined by a mass to, a spring constant k, and a damping 
coefficient c: 

nix = —kx — cx + v (22) 

A nominal control law for the system is represented by the 
following proportional feedback equation: 

v* = k p ■ (x — x c ) (23) 

where x and x c represent an actual and a commanded actua- 
tor position respectively, v and v* represent the net actuator 
output force and that commanded by the nominal control law 
respectively, and k p is a gain coefficient within the nominal 
controller. Diagrams of the mass-spring-damper system and 
the proposed control law are shown in Figure 2. Values for 
all variables in the controlled mass-spring-damper system are 
given in Table 1 . 

In this example, the net output force exerted by the controlled 
effectors on the environment and the performance metric used 
to regulate that output are both one dimensional: 

v = pv* (24) 

Figure 3 shows the behavior of the system as p is linearly var- 
ied over the domain [0.2, 1], It should generally be expected 
that smaller values of p will induce greater errors in tracking 
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Plant Model 


v(k) r\V*(k) 


C 


| p(k) 

Control System 


e{k)r ^x(k) 
x c (k) 


Figure 2. Diagram of mass-spring-damper plant model and 
control system 


Coefficient 

Value 

m 

1 

k 

2 

c 

5 

kp 

15 


Table 1. Coefficients used in mass-spring-damper simulation 


a desired output profile, and these induced errors will result 
in greater net actuator output force demands by the nominal 
control law. Both of these features are clearly visible in the 
experimental results given in Figure 3. Another observation 
that can be made from the results given in Figure 3 is that 
while p < 1 will always result in an apparent reduction in the 
net output control effort that would have been commanded by 
a nominal control law at a given control time-step, the error 
dynamics that are induced by lowering p will not necessarily 
result in lowered net system output control effort over a finite 
window of observation. A marginally increasing trend in the 
peak-to-peak actuator loads over a cycle is observed as p is 
lowered; this type of behavior illustrates the fact that achiev- 
ing a good tradeoff between the reduction of component load- 
ing and induced errors in trajectory tracking will generally 
require closed loop regulation of p. 

3. The Prognostics-Based Control Problem 

The prognostics-based control problem can generally be 
viewed as an optimization problem, in which implemented 
control routines will select values of pi at each control time- 
step in an attempt to minimize the risks posed by the appli- 
cation of load to degrading components, while also minimiz- 
ing any deviation from a system’s nominal performance. The 
problem of specifying appropriate metrics for assessing the 
risk posed by probabilistic prognostic predictions of future 
component health deterioration may generally be considered 
independently from the problem of building prognostic mod- 
els. Assuming that discrete Markov modeling will provide 
a sufficiently accurate approximation of a system’s fault dy- 
namics, then the Markov process notation described in Sec- 
tion 2 could be used to evaluate any given risk metric for use 
in any given prognostics-based control method. Future work 
will address the specification of appropriate risk metrics and 
the use of this Markov process formulation for deriving and 
evaluating prognostics-based control policies on sample ap- 
plications. 


Preformance Factor Profile 




Time (minutes) 


Figure 3. Plots showing position profile tracking and actuator 
loading dynamics as p is varied over the range [0.2, 1] 


If aversion to the potential degradation of a system’s nomi- 
nal output loading performance and aversion to the potential 
degradation of component SOH are adequately expressed in 
terms of an expectation of accumulated state transition costs, 
then the search for a control policy that optimizes a stochas- 
tic system of the form disclosed in Eq. (19) is expressible 
as a Markov decision process (MDP). MDPs are commonly 
used to analyze problems involving decision making in the 
presence of uncertain or stochastic information, and opti- 
mizing control policies may be identified using well studied 
MDP optimization techniques such as backwards induction 
for finite horizon optimization problems, and linear program- 
ming, value iteration, and policy iteration for discounted and 
average-reward infinite horizon optimization problems. The 
requirement that fault risk must be expressed as an expecta- 
tion of accumulated state transition costs over a finite or in- 
finite horizon may seem to be an overly restrictive constraint 
on the general problem of quantifying risk; however, over the 
past several decades much has been published on the theory 
of encoding various forms of risk aversion into the specifi- 
cation of MDP state transition costs (Hernandez & Marcus, 
1996; Ruszczyriski, 2009). 

4. Conclusions 

This paper introduced a novel Markov process representation 
of component health dynamics. A metric representing the 
relative deviation between instantaneous samples of the net 
output force that would have been enacted by a nominally 
controlled system, and the net system outputs that are actually 
enacted by an implemented prognostics-based control routine 
at each time-step, was used to define the action space of the 
Markov process. The state space of the proposed Markov pro- 
cess formulation was defined in terms of an abstracted met- 
ric representing the relative health remaining in each of the 
system’s components. Operational constraints on minimum 
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acceptable system output performance and maximum accept- 
able fault growth risk were formalized, and the mappings nec- 
essary to impose those constraints on the domain of feasible 
control actions available to implemented prognostics -based 
control routines were specified. Arguments were provided 
for the potential convenience and robustness of the proposed 
notation for evaluating prognostics-based controllers. 
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